Full-Memory Transformer for Image Captioning

The Transformer-based approach represents the state-of-the-art in image captioning. However, existing studies have shown Transformer has a problem that irrelevant tokens with overlapping neighbors incorrectly attend to each other relatively large attention scores. We believe this limitation is due incompleteness of Self-Attention Network (SAN) and Feed-Forward (FFN). To solve problem, we present Full-Memory method for improves performance both encoding language decoding. In step, propose Full-LN symmetric structure, which enables stable training better model generalization by symmetrically embedding Layer Normalization on sides SAN FFN. decoding Memory Attention (MAN), extends traditional mechanism determine correlation between results input sequences, guiding focus words need be attended to. Our evaluated MS COCO dataset achieves good performance, improving result terms BLEU-4 from 38.4 39.3.